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DOCUMENT-IDENTIFIER: US 5898434 A 

TITLE: User interface system having programmable user interface elements 
Brief Summary Text (6) : 

The Macintosh computer user interface employs what is referred to as a desktop 
metaphor for the basis of its user interface. Through use of a mouse or trackball 
in communication with the computer, the user can select and/or move certain conic 
objects on the screen of the computer to control its operation. The desktop 
metaphor refers to the fact that: (1) the screen of the Macintosh computer often 
looks somewhat like the blotter of a desktop, with a menu bar across the top of the 
blotter from which the user can select various control features; (2) there is a 
trash can icon in the lower right hand corner of the blotter which can be used to 
remove items from the computer; and (3) programs and documents created with the 
programs are represented by iconic images of pages which can be placed in iconic 
images of folders and arranged on the blotter for easy access and organization, 
like the files of a filing cabinet in an office. These icons can be arranged- on 
the screen in a number of different manners, such as according to their type, size 
and color, and the user can search for files by searching for words listed in the 
title bar associated with each icon, but the user cannot search for a specific 
iconic image itself. 

Brief Summary Text ( 7 ) : 

Another popular portion of the Macintosh user interface is the window feature. When 
a user selects an icon of a floppy disk or hard disk in a certain manner, a window 
(a rectangular shaped box which allows the user to "see" the information stored in 
the drive) appears on the screen of the computer. This window typically contains a 
number of folder, program and document icons and other images which represent the 
information stored in the drive. If a user selects a folder icon in a certain 
manner, the folder will "open" into another window representing the information 
within that folder. As previously stated, document and program icons can be placed 
i n folder icons, but not vice versa, and no icon can be placed directly on a 
document. Iconic images have become an important feature of many user interfaces 
because they can be used to represent different types of data in a form which is 
readily understandable to the user. Icons, however, have had little functionality 
associated with them because they are typically only used to represent stored 
information that a user can access fairly directly, i.e., by selecting the icon. 
But some icon-like images have had some functionality associated with them, such as 
the button-like images that can be created with the HyperCard software application 
sold by Claris Corporation of Mountain View, Calif. The HyperCrad application 
allows users to create card-like images that can have various images and features 
associated with them. 

Brief Summary Text (15) : 

A preferred embodiment of the present invention comprises a user interface system 
having a plurality of user interface elements for marking, finding, organizing, and 
processing data within documents stored in an associated computer system. Each 
element typically has an appearance which is uniquely related to the data or the 
function the element is designed to represent or perform, respectively. In their 
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simplest form, these elements are only used to mark data within a document. Each 
element, however, can also be programmed to cause the computer to perform some 
function in association with the marked data, such as printing the data or mailing 
the data to someone. A user can select particular data within a document using an 
element and have that data associated with the element in memory. Data marked with 
common elements can be found by searching for a particular representative element 
in memory. Users can create their own elements, program elements with their own 
desired functionality, and modify existing elements. Elements can also be 
compounded together so as to cause a combination of tasks to be performed by simply 
activating one element. Illustrative uses for and functionality of elements in 
specialized applications such as a computerized camera system, a remote control 
device and a portable multimedia player are also disclosed. Additionally, a method 
and apparatus for marking and searching for data, in systems having limited 
input/output capability, is also disclosed. 

Drawing Description Text (11): 

FIG. 6c illustrates a technique for labeling a folder of data on a display screen 
with one or more user interface elements. 

Drawing Description Text (20) : 

FIG. 10b illustrates the operation of a link element and a dialog box generated by 
activation of the link element. 

Detailed Description Text (6) : 

A collection of these elements is shown in FIG. 2. In the preferred embodiment of 
the present invention, each element has a fanciful appearance and a programmed 
functional characteristic that allows that element to be used to mark, find, 
organize and process data within documents stored within the computer 2 in a manner 
which is unique to the appearance of that element. It should noted, however, that 
the association between the unique appearance of an element and its programmed 
function is not necessary, and that the elements need not have any appearance at 
all, i.e., aural representations could be used instead of visual representations to 
communicate the different types of elements to a user who may not be able to see 
the display screen 14 . 

Detaile d Descri ption Text (15) : 

With element bases being produced at a number of different sources, many uniquely 
pleasing and functional element bases and element instances could be created. For 
example, Project element 42 was designed to impart a leather-based, ink-stamped 
look, with its speckled rectangular boarder. Other interesting elements are the 
Rush element 4 4 and the Lightbulb element 46. The Rush element 4 4 could be 
singularly used to mark a computer generated document as a rush document, or it 
could be combined with another element, such as To Do element 40, to indicate that 
something that is to be done, is to be done immediately. The Lightbulb element 4 6 
could likewise be used to decorate the appearance of a document, such as to 
indicate a good idea, or it could be used to indicate that something is to be 
remembered. It should be noted that within the context of the present invention, a 
"document" includes any electronically generated image for creating and editing 
data displayed therein, such as a page of text, a spreadsheet page, an electronic 
mail message screen, a video image or photo, and a folder of the type illustrated 
in FIG. 6c, but does not include a standard window within which typical iconic 
images are stored and organized, such as the program, folder and document icons 
mentioned in the discussion of the prior art above. 

Detailed Description Text (16) : 

FIG. 3 also includes the Confidential element 48, which could be used to restrict 
access to documents. For example, when a document was marked with the confidential 
element, only users who entered a password would be able to read the document. 
Other examples of how the various elements of FIG. 3 could be utilized include: (1) 
documents marked with the Sent element 50 could cause the document to be 
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automatically dated when sent to somebody; (2) documents marked with the Received 
element 52 could cause the document to be automatically dated when it was received 
by somebody; (3) documents marked with the Internal Report element 54 could be 
restricted to distribution only within a company or an office; (4) documents marked 
with the Draft element 56 would only print with the word "draft" written across 
each page; and (5) documents marked with the Link element 58 could be automatically 
linked to another document or another element located somewhere else in the memory 
of the computer 2. 

Detailed Description Text (23) : 

Element instances can also be associated with entire pages of data in a computer, 
as is illustrated in FIG. 6a, distinguishable portions of data on a page of data, 
as is illustrated in FIG. 6b, or folders containing multiple pages of data and 
other types of information, as is illustrated in FIG. 6c. FIG. 6a illustrates a 
technique whereby different pages of data on display screen 14 are marked with 
elements, thereby associating each page with any element marking that page. Mark 
element instances 96 could be placed at any position desired on page 98, but would 
preferably be placed on page 98 so that they would still be somewhat visible if 
page 98 was covered by another page at a later time, as is demonstrated by the 
other visible portions of Mark element instance 96 on pages behind page 98. As 
shown in FIG. 6a, all of the Mark element instances 96 have identical element 
attributes, they are all numbered "10", but could also be assigned different 
numbers, as desired by the user, to further distinguish each element instance. 

Detailed Description Text (25) : 

FIG. 6c illustrates that element instances can be used to mark folders of data on a 
display screen. Computer 2 has an image of a folder 108 displayed within display 
screen 14. Folder 108 could both display data, such as the notes 11 0 pictured on 
the front of folder 108, and act as a storage for other information which relates 
to the marks associated with the folder 108. In FIG. 6c, folder 108 is marked with 
two element instances, the Twin Peaks element instance 112, which relates to the 
project the user is working on, and the To Do element instance 40, which indicates 
that certain documents within the folder 108 are marked with the To Do element 
instance 40. This feature of marking folders with element instances is useful 
because by simply marking a folder with an element instance, such as Twin Peaks 
instance 112, all of the documents containing the Twin Peaks instance 112 would be 
collected within folder 108 . 

Detailed Description Text (27) : 

As described above, the various attributes of an element can be used to find data 
stored in the computer 2 . One technique for querying the element database of 
computer 2, is shown in FIG. 7a, which illustrates a folder tool for filtering 
through stored instances to find data that matches the query constructed by the 
user. By selecting the Find element 104 in the well 16 of display screen 14 and 
moving an instance 116 of element 104 into the display screen, the user causes a 
finder tool box 118 to be opened on the display screen. To find particular data, 
the user need only select an instance of that element from the well 16, such as 
Phone element instance 120, and drop the instance into one of the query boxes 122 
of the finder tool box 118. The processor 6 of computer 2 would then proceed to 
filter through the database in its memory 8 until it had found all of the documents 
or pages containing the Phone instance 120, which acts as a proxy to the data 
stored in memory 8. 

Detailed Description Text (29) : 

More sophisticated filter queries could also be performed by computer 2 if 
additional features were added to finder tool box 118, such as a term search 
section. The performance of this feature, however, would be highly dependent on 
whether the user had the ability to enter recognized text. If the computer 2 is 
capable of having text characters entered, such as through a keyboard, then the 
computer could easily search for matching sets of text characters within the data 
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associated to each queried element. If the user could not enter text in this 
fashion, the user may still be able to copy text from the display screen 14 and 
paste the copied text into the term search section. Of course, this would only work 
if the text being searched for exactly matched the text stored in the term search 
section. In addition, it should also be noted that the user could perform 
hierarchical searches through use of the finder tool box 118 illustrated in FIG. 7a 
by simply placing additional element instances in the remaining query boxes 122. 
Hence, the computer 2 would first filter through all of the data to locate data 
associated with Phone element instance 120, and then filter through that data to 
find data associated with the next element instance, etc., until all of the queried 
boxes 122 had been processed. 

Detailed Description Text (38) : 

Link element 172 of FIG. 10b, which is the functionally the same as link element 58 
of FIG. 2, only different in appearance, is also an active element. When an 
instance of link element 172 is marked on a page, link dialog box 174 appears. Link 
dialog box 174 will then stay visible on the display screen 14 while the user pages 
through the various pages of information stored in computer 2. When a desired page 
is located, the user then selects the "this pages" button 176 causing an instance 
of the link element 174 to be marked on the desired page and causing the computer 2 
bounce back and display the original page on which the instance of the link element 
172 was originally attached. To jump between the linked pages in the future, the 
user would only need to select the instance of the link element 172. 

Detailed Description Text (51) : 

When the user has entered the attributes for the compounded instances, all of the 
instances but the primary instances are hidden from view until the next time the 
primary instance is selected, thereby allowing the user to perform more operations 
and store more information within the area of the display normally occupied by a 
single instance. In addition, when the performance of the function of a primary 
instance is triggered, the other instances compounded to the primary instance 
perform their functions as well in a hierarchical order. 

Detailed Description Text (60) : 

As described herein, the element serves as a proxy for the marked data in a 
preferred embodiment. Typically this is accomplished by creating a separate data 
structure which serves as an index or look-up table comprising identifying 
information to identify the element to the system, and comprising information to 
identify the location in memory of the marked data. Additionally, the data 
structure comprises information to identify the location in memory where user 
entered attributes for each instance of the element have been stored. In a 
preferred embodiment, the information identifying the element to the system may be 
used to determine the system attributes of the element. Of course, if desired, any 
of this information may be stored along with the marked data. For example, the 
index or look-up structure may identify all the data marked with a given instance 
of an element, and the user attributes may be entered along with the data so 
marked. Finally, if desired, an index or look-up table need not be supplied, and 
the identifying information for the element may be stored in the same data 
structure as the marked data. Then, when a user desires to search for instances of 
an element, the data structure may be searched for the information identifying the 
element being searched for. As a further alternative, a separate data structure may 
be created comprising information identifying the element. Each time an instance is 
applied to data, the marked data can then be replicated and stored in the separate 
data structure along t with the instance. Numerous other methods of identifying the 
element to the system, and of associating each instance with data marked with the 
instance, and system and user attributes of the element may be envisioned by those 
skilled in the art. 

Detailed Description Text (72) : 

Referring to FIG. 21, a further device upon which physical switches may be used, is 
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shown. Illustrated is remote control device 2100, which may be, for example, a 
remote control device which controls any type of computerized system. For example, 
in one embodiment remote control device 2100 is used to control operation of a 
computer system having the capability of playing, recording, and storing audio 
data. The audio data may include messages from a voice mail system, individual 
voice notes, any other audio data, or may comprise textual messages such as 
electronic mail messages which are converted to audio data by a voice synthesizer. 
As shown, remote control device 2100 comprises play button 2101, stop button 2102, 
fast forward 2103, and rewind button 2104. Additionally, if desired, remote control 
device 2100 may also comprise a microphone and a record button. Also shown are 
buttons 2110, 2111, 2112, and 2113. Each of these buttons may be used to apply a 
user interface element to any of the audio documents referred to above. 

Detailed Description Text (74): 

If the user desires to filter through the audio documents stored in the computer 
system associated with remote control device 2100, the user simply pushes the lower 
half of one or more of buttons 2110, 2111, 2112, and 2113, as desired. 
Specifically, if the user wishes to filter, for example, for audio documents that 
have been marked with the user interface elements associated with buttons 211 0 and 
2112 the user simply presses the lower half of buttons 2110 and 2112. Once the 
lower half of any of the buttons 2110, 2111, 2112, and 2113 has been pressed, the 
system is in filter mode. As shown, four LED ' s 2115, 2116, 2117, and 2118 are 
disposed below the buttons 2110, 2111, 2112, and 2113, respectively, to indicate 
that the system is in filter mode. In the example described just above, where the 
system is set to filter for audio documents marked with the user interface elements 
associated with buttons 2110 and 2112, LED 1 s 2115 and 2117 will be on, while LED's 
2116 and 2118 will be off. While the system is in filter mode, the user can search 
for audio documents having the selected elements by pressing fast forward button 
2103 to get to the next audio document in memory which has been associated with the 
user interface elements corresponding to the lit LED 1 s , or by pressing rewind 
button 2104 to get to the closest previous audio document having the selected 
elements . 

Detailed Description Text (92) : 

In the present invention, various types of data or software may be supplied with 
user interface elements pre-applied to appropriate portions. For example, an 
encyclopedia may mark portions of data with one or more types of elements. Thus, 
for example, a biography giving basic information about an individual may be marked 
with an element having, for example, the appearance of several persons, to indicate 
that information on contemporaries of the individual in related fields of endeavor 
may be obtained by selecting the element. Dates within the biography may contain 
elements having, e.g., the appearance of a calendar which, upon selection, provide 
information of related events occurring around the time of the date in the main 
text. Similarly, numerous other types of pre-supplied elements may be envisioned. 
Note that in addition to providing a convenient link between two portions of data, 
the user interface element, by its appearance, conveys to the user the type of 
information which will be provided upon selecting the element. Further, the pre- 
supplied elements may contain functionality in addition to information. For 
example, the supplier of a catalog may pre-supply order elements. Further, the 
order elements may be context sensitive with the specific functionality, and/or 
information provided depending upon the portion of the document with which the 
element is associated. 
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